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In  1975  Department  of  Defense  Management  by  Objective  MBO  9-2  directei 
each  service  to  provide  the  means  for  identifying  and  providing  visibility 


of  the  operating  and  support  (O&S)  costs  of  its  major  weapons  systems.  The 
vehicle  for  accomplishing  this  objective  is  referred  to  as  the  Visibility 
and  Management  of  Operating  and  Support  Costs  (VAMOSC)  program.  Through 
this  program,  the  Air  Force  developed  its  VAMOSC  system,  a  comprehensive 
management  information  system. 

The  Air  Force  VAMOSC  system  is  designed  to  collect  and  display  O&S 
costs  at  MDS  (Mission-Design-Series)  and  component  levels  of  aircraft  and 
also  at  TMS  (Type-Model-Series)  and  component  levels  of  ground  communica- 
tions-electronics  equipment.  It  consists  of  three  subsystems: 

(1)  WSSC  (D160),  which  deals  with  aircraft, 

(2)  C-E  (D160A) ,  which  deals  with  ground  communications-electronics 


equipment, 

and  (3)  CSCS  (D160B) ,  which  deals  with  subsystems  and  components. 

Because  the  data  provided  by  VAMOSC  will  be  used  to  support  decisions 
involving  large  financial  commitments,  the  accuracy  of  this  data  and  the 
validity  of  the  VAMOSC  logic  are  of  prime  importance.  Desmatics,  Inc., 
under  Air  Force  Contract  No.  F33600-80-C-0554,  is  currently  engaged  in  a 
technical  effort  to  validate  the  WSSC  subsystem  of  VAMOSC.  This  paper  ad¬ 
dresses  that  validation  effort.  Although  the  paper  focuses  on  WSSC,  the 
general  concepts  are  also  applicable  to  the  other  two  VAMOSC  subsystems. 
Because  the  validation  effort  is  ongoing,  any  conclusions  in  this  paper 


must  be  regarded  as  preliminary. 
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The  WSSC  Subsystem 

WSSC  produces  two  annual  reports  which  list  O&S  costs  for  each  MDS. 
These  reports,  which  differ  primarily  in  format,  are  termed  the  "DOD/CAIG 
report"  and  the  "USAF  detail  report."  The  former  report  is  designed  to 
satisfy  the  requirements  specified  by  the  DOD  Cost  Analysis  Improvement 
Group  (CAIG)^.  The  latter  report  provides  additional  details  that  are 
useful  to  various  users  within  the  Air  Force.  Other  special  interrogation 
reports  are  available  by  MDS,  command,  and  base. 

A  major  problem  faced  by  WSSC  is  that  most  available  cost  data  is 
not  identified  directly  to  an  MDS.  Most  of  the  data  sources  used  by  WSSC 
are  functional  data  systems  that  do  not  relate  consumption  of  resources  to 
weapons  systems.  Of  course,  in  the  case  of  indirect  costs,  such  as  instal¬ 
lation  support  costs ,  there  is  no  way  to  identify  them  by  MDS. 

Because  of  the  lack  of  a  one-to-one  relationship  between  costs  and 
MDS,  much  of  the  cost  data  must  be  allocated.  It  is  necessary  for  WSSC  to 
allocate  not  only  the  typical  "overhead"  types  of  costs,  but  also  direct 
costs  which  the  input  data  systems  do  not  identify  by  MDS.  WSSC  uses  a 
series  of  algorithms  that  relate  functional  costs  to  measures  of  activity 
in  order  to  allocate  the  costs  to  the  weapons  systems. 

In  WSSC  four  general  quantities  are  used  to  distribute  costs:  (a) 

personnel  strengths;  (b)  the  inventory  of  a  particular  MDS  as  measured  by 
2 

possessed  hours  ;  (c)  the  number  of  flying  hours  for  a  particular  MDS;  and 

^"Aircraft  Operating  and  Support  Cost  Development  Guide,"  Cost  Analysis  Im¬ 
provement  Group,  Office  of  the  Secretary  of  Defense,  1980. 

2 

An  aircraft  may  be  "possessed"  by  different  commands  or  bases  during  the 
year.  Thus,  the  8760  hours  in  the  year  are  used  to  determine  the  inventory 
of  an  MDS  for  each  command  and  base  during  the  year. 
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(d)  reported  maintenance  direct  labor  hours.  In  addition,  estimated  cost 
factors  (such  as  annual  medical  care  costs  per  person)  are  used  in  cost 
allocation. 
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Validation  of  WSSC:  An  Overview 


The  reports  produced  by  WSSC  serve  as  an  important  management  tool  for 
providing  visibility  of  weapons  systems  operating  and  support  costs.  The 
extent  to  which  WSSC  cost  outputs  are  satisfactory  for  their  intended  pur¬ 
pose  depends  upon  two  components:  (1)  the  soundness  of  system  design  and 
(2)  the  accuracy  of  input  data.  The  accuracy  of  VAMOSC's  output  and  the 
validity  of  its  logic  are  paramount  to  the  utility  of  the  system  to  the  user. 
For  this  reason,  the  Air  Force  Office  of  VAMOSC  is  relying  on  Independent 
,  contractors  to  validate  the  VAMOSC  logic  and  data  sources. 

The  overall  validation  of  the  WSSC  system  may  be  viewed  as  the  three- 
phase  process  indicated  in  Figure  1: 

(1)  Phase  I  focuses  on  the  allocation  algorithms  and  data  selection 
logic  comprising  WSSC, 

(2)  Phase  II  addresses  the  accuracy  of  the  WSSC  inputs  relative  to 
the  books  of  original  entry, 

and  (3)  Phase  III  examines  the  accuracy  of  the  books  of  original  entry 
relative  to  what  actually  occurred. 

Phase  I  validation,  which  addresses  the  system  design,  is  Desmatics' 
task  under  its  Air  Force  contract.  Actually,  Phase  I  validation  concerns 
itself  with  two  types  of  validation:  internal  validation  and  external 
validation.  Often  internal  validation  is  referred  to  as  "verification," 
i.e.,  verifying  that  the  system  performs  as  it  was  designed.  Thus,  Internal 
validation  primarily  involves  debugging  of  the  associated  computer  programs. 
Similarly,  external  validation  is  often  referred  to  simply  as  "validation," 
and  addresses  whether  the  system  is  designed  as  it  should  be.  External 


VALIDATION  OF  THE  WSSC  SYSTEM 


Figure  1:  The  Three-Phase  Validation  Process 
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validation  critically  examines  system  logic,  input  data  selection  procedures, 
and  output  data. 

For  the  most  part.  Internal  validation  has  been  accomplished  by  the 
Office  of  VAMOSC  in  its  production  test  of  WSSC.  Thus,  Desmatlcs  is  pri¬ 
marily  concerned  with  external  validation.  It  should  be  stressed,  however, 
that  the  overall  validation  task  consists  of  a  large  degree  of  interaction 
and  cooperation  between  Desmatlcs  and  the  Office  of  VAMOSC.  This  synergis¬ 
tic  relationship  has  had  a  positive  impact  on  the  WSSC  system. 
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Indirect  Validation 

The  best  method  of  validation  is,  of  course,  a  direct  one — that  is, 
one  in  which  the  WSSC  cost  allocations  are  compared  with  the  true  costs. 
Unfortunately,  there  is  no  way  to  validate  the  WSSC  system  directly,  be¬ 
cause  the  true  costs  are  unknown.  Thus,  any  validation  must  be  of  an  in¬ 
direct  nature. 

Indirect  validation  approaches  may  be  classified  as  dealing  with  two 
major  types  of  validity:  (1)  face  validity  and  (2)  mathematical  validity. 

The  former  is  concerned  with  qualitative  assessment,  while  the  latter  deals 
with  quantitative  assessment.  Our  validation  of  WSSC  addresses  both  of 
these  validity  components. 

Face  validity,  in  essence,  is  the  examination  of  the  WSSC  system  on 
a  subjective,  common-sense  basis.  In  conducting  this  qualitative  assess¬ 
ment  of  WSSC,  Desmatics  focused  on  whether  the  algorithms  provide  equitable 
allocations,  so  that  no  group  of  MDS's  bears  an  Inordinate  amount  of  the 
allocated  costs.  Somewhat  related  was  consideration  of  the  allocation  frame¬ 
work  to  determine  whether  alternative  allocation  methods  might  provide  better 
results. 

A  major  check  on  face  validity  is  to  determine  whether  potential  users 
have  any  suggestions  for  modifications  to  the  system.  Because  such  sugges¬ 
tions  often  result  in  major  improvements,  the  Office  of  VAMOSC  has  worked 
closely  with  potential  users  of  WSSC  during  its  development. 

In  parallel  with  qualitative  face  validity,  Desmatics  is  also  examining 
the  WSSC  system  from  a  quantitative  standpoint.  This  involves  evaluation  of 
the  mathematical  framework  of  the  system  (including  assessment  of  the  alloca¬ 
tion  logic),  consideration  of  algorithm  sensitivity,  and  examination  of  pos- 
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sible  data  discrepancies.  The  following  sections  provide  a  brief  discus¬ 
sion  of  some  aspects  of  the  ongoing  quantitative  evaluation. 
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Sensitivity  Analysis 

A  number  of  FY81  costs  within  WSSC  were  allocated  on  the  basis  of 
flying  hours  (FH)  and  possessed  hours  (FH) ,  combined  into  an  allocation 
ratio.  However,  just  how  FH  and  PH  should  be  weighted  within  that  ratio 
is  an  open  question.  To  determine  the  impact  the  weighting  has  on  the 
resulting  cost  allocations,  a  sensitivity  analysis  may  be  conducted  within 
each  cost  category  which  uses  flying  hours  and  possessed  hours  as  alloca¬ 
tion  variables. 

Sensitivity  of  the  WSSC  results  may  be  examined  at  different  levels 
of  allocation.  For  example,  WSSC  not  only  allocates  costs  to  the  MDS  level, 
but  also  to  the  command-base-MDS  level.  Of  course,  the  amount  allocated  to 
a  particular  MDS  is  equal  to  the  sum  of  the  amounts  allocated  to  all  command- 
base-MDS  combinations  involving  that  particular  MDS. 

In  the  cases  involving  allocation  ratios  based  on  FH  and  PH,  sensitivity 
can  be  measured  by  examining  the  differences  in  allocated  costs  that  result 
from  using  only  flying  hours  as  opposed  to  using  only  possessed  hours.  There 
are  three  measures  that  may  prove  useful  in  assessing  the  sensitivity  of  the 
cost  allocations  to  the  use  of  flying  hours,  possessed  hours,  or  a  mixture 
of  both.  One  is  an  absolute  measure;  the  other  two  are  relative  measures. 

Consider  a  specific  O&S  cost  category  for  which  costs  are  allocated 
by  means  of  an  allocation  ratio  involving  flying  hours  (FH)  and  possessed 
hours  (PH).  Regardless  of  the  level  of  allocation,  we  can  define 

Absolute  Sensitivity  ■  A$ 

■  $(FH  only)  -  $(PH  only).  (1) 


That  is,  we  are  measuring  the  change  in  dollar  amount  (either  positive  or 


negative)  that  would  result  from  using  only  flying  hours  as  an  allocation 
basis  instead  of  using  only  possessed  hours.  If,  for  example,  we  are  con¬ 
sidering  the  command-base-MDS  level  allocation,  we  would  have  a  set  of  435 
sensitivity  observations  since  there  is  a  total  of  435  command-base-MDS 
combinations. 

In  addition  to  (1) ,  two  relative  sensitivity  measures  provide  useful 
information.  These  are: 

Relative  Sensitivity  #1  *  A$/Total  Cost  Category  $ 

and 

Relative  Sensitivity  #2  *  A$/Total  O&S  $ 

Measure  #1  yields  an  indication  of  sensitivity  of  the  measures  for  one  com¬ 
mand-base-MDS  observation  relative  to  the  total  cost  category  dollars  to  be 
allocated,  while  measure  #2  provides  a  similar  indication  relative  to  the 
total  O&S  amount  for  all  command-base-MDS  combinations. 

As  an  example.  Figure  2  presents,  for  the  general  depo^.  support  cost 
category,  the  distribution  of  the  observed  absolute  sensitivity  measures 
for  each  of  the  435  command-base-MDS  combinations.  As  can  be  seen,  the  dis¬ 
tribution  is  highly  skewed  to  the  right,  which  indicates  that  a  number  of 
command-base-MDS  combinations  will  be  allocated  a  relatively  large  additional 
amount  of  general  depot  support  costs  as  weighting  in  the  allocation  ratio 
shifts  from  possessed  hours  to  flying  hours. 

The  maximum  observed  value  in  this  set  of  data  is  $2.56M.  Relative  to 
the  total  general  depot  costs  of  $246. 1M  and  to  total  O&S  costs  of  $11,048.8M, 
this  is  1.04%  and  0.02%,  respectively. 

Results  such  as  these,  which  provide  absolute  and  relative  differences 
in  cost  allocations,  may  be  used  in  conjunction  with  the  resuits  from  other 
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SENSITIVITY  (GENERAL  DEPOT  SUPPORT) 
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Figure  2:  Distribution  of  Absolute  Sensitivity  Measures 
for  General  Depot  Support  Costs 
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cost  categories  to  examine  the  overall  sensitivity  of  cost  allocations  to  the 
use  of  flying  hours  versus  possessed  hours.  Based  on  this  examination,  a 
judgment  may  be  made  about  the  criticality  of  determining  the  most  appropri¬ 
ate  weighting  of  flying  hours  and  possessed  hours.  If  the  overall  sensitivity 
were  relatively  small,  for  example,  a  "ball-park"  estimate  of  the  weighting 
might  suffice.  However,  if  the  sensitivity  were  quite  large,  further  invest¬ 
ment  of  effort  aimed  at  homing  in  on  the  optimal  weighting  might  be  indicated. 
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Examination  of  Apparent  Discrepancies 


To  provide  a  quantitative  check  on  the  WSSC  results,  a  number  of  sta¬ 
tistical  procedures  can  be  used.  Including,  of  course,  the  calculation  of 
summary  statistics  such  as  means  and  standard  deviations.  Often  graphical 
presentations  are  helpful  in  indicating  where  there  are  possible  problems 
within  the  WSSC  logic,  because  "a  picture  is  worth  a  thousand  words"  (or 
numbers) . 

Figure  3  depicts  a  simplified,  somewhat  diagrammatic,  plot  of  one  com¬ 
ponent  of  installation  support  costs  (RPM  airman  pay  and  allowances)  at  a 
base  against  the  number  of  supported  people  at  that  base.  Each  observation 
in  the  plot  represents  one  base.  As  can  be  seen,  there  are  a  few  suspicious 
data  points:  data  point  A  and  the  clump  of  data  points  labeled  B.  Outliers 
such  as  these  provide  a  strong  indication  that  something  is  wrong  somewhere. 

In  this  case,  it  has  been  determined  that  the  existence  of  data  point 
A  reveals  a  problem  with  the  input  data  that  must  be  taken  into  account  by 
WSSC.  Within  the  Alaskan  Air  Command,  all  installation  support  costs  for 
its  three  bases  are  reported  in  the  accounting  system  at  only  one  base 
(Elmendorf) ,  while  the  strengths  are  reported  to  the  appropriate  bases. 

Thus,  the  Elmendorf  installation  costs  appear  much  out  of  line  relative  to 
the  base  population.  In  tracking  down  the  second  apparent  discrepancy,  it 
turned  out  that  data  clump  B is  explained  by  the  fact  that  these  five  bases 
are  ones  at  which  depots  are  located,  and  a  larger  share  of  RPM  at  depots 
is  done  by  civilians. 
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Summary 


This  paper  has  provided  a  brief  discussion  of  the  WSSC  system  vali¬ 
dation,  concentrating  on  the  current  research  of  Desmatics,  Inc.  That 
research  effort,  which  deals  with  Phase  I  validation,  extends  through 
September  1983.  Because  there  is  no  way  in  which  to  validate  WSSC  di¬ 
rectly,  Desmatics  is  using  an  indirect  approach  comprised  of  a  qualitative 
assessment  (face  validity)  and  a  quantitative  assessment  (mathematical 
validity) . 

To  date,  the  major  portion  of  the  Desmatics  work  has  concentrated  on 
the  qualitative  evaluation  and  on  setting  up  the  framework  for  the  quanti¬ 
tative  evaluation  effort.  Although  this  paper  has  discussed  some  aspects 
of  the  quantitative  evaluation,  a  major  portion  of  that  evaluation  will  be 
completed  after  the  FY82  WSSC  data  becomes  available  early  in  1983.  None¬ 
theless,  a  number  of  recommendations  have  already  been  made.  For  example, 
Desmatics  has  suggested: 

(1)  investigating  the  application  of  regression  techniques  to  sepa¬ 
rate  fixed  and  variable  costs  in  certain  cost  categories, 

(2)  replacing  allocations  based  on  flying  hours  and  possessed  hours 
with  ones  based  on  personnel  strength  ratios  in  some  algorithms, 

(3)  revising  some  personnel  strength  allocation  ratios  to  make  them 
more  appropriate, 

(4)  providing  separate  reporting  of  direct  and  overhead  maintenance 


COSti , 

and  (5)  using  a  one-stage  allocation  as  opposed  to  a  two-stage  allocation 
in  some  algorithms. 
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